Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory. In this paper, we identify that fine-tuning performance is significantly impacted by hyper-parameter choices. We examine various key hyper-parameters and empirically evaluate their impact in fine-tuning CLIP for classification tasks through a comprehensive study. We find that the fine-tuning performance of CLIP is substantially underestimated. Equipped with hyper-parameter refinement, we demonstrate CLIP itself is better or at least competitive in fine-tuning compared with large-scale supervised pre-training approaches or latest works that use CLIP as prediction targets in Masked Image Modeling. Specifically, CLIP ViT-Base/16 and CLIP ViT-Large/14 can achieve 85.7%,88.0% finetuning Top-1 accuracy on the ImageNet-1K dataset . These observations challenge the conventional conclusion that CLIP is not suitable for fine-tuning, and motivate us to rethink recently proposed improvements based on CLIP. We will release our code publicly at \url{https://github.com/LightDXY/FT-CLIP}.

Copy-Paste is a simple and effective data augmentation strategy for instance segmentation. By randomly pasting object instances onto new background images, it creates new training data for free and significantly boosts the segmentation performance, especially for rare object categories. Although diverse, high-quality object instances used in Copy-Paste result in more performance gain, previous works utilize object instances either from human-annotated instance segmentation datasets or rendered from 3D object models, and both approaches are too expensive to scale up to obtain good diversity. In this paper, we revisit Copy-Paste at scale with the power of newly emerged zero-shot recognition models (e.g., CLIP) and text2image models (e.g., StableDiffusion). We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable. To make such success happen, we design a data acquisition and processing framework, dubbed "X-Paste", upon which a systematic study is conducted. On the LVIS dataset, X-Paste provides impressive improvements over the strong baseline CenterNet2 with Swin-L as the backbone. Specifically, it archives +2.6 box AP and +2.1 mask AP gains on all classes and even more significant gains with +6.8 box AP +6.5 mask AP on long-tail classes.

Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmative answers leveraging two instantiations of \emph{group-wise clipping}. To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization. This results in private learning that is as memory-efficient and almost as fast per training update as non-private learning for many workflows of interest. While per-layer clipping with constant thresholds tends to underperform standard flat clipping, per-layer clipping with adaptive thresholds matches or outperforms flat clipping under given training epoch constraints, hence attaining similar or better task performance within less wall time. To explore the limits of scaling (pretrained) models in differentially private deep learning, we privately fine-tune the 175 billion-parameter GPT-3. We bypass scaling challenges associated with clipping gradients that are distributed across multiple devices with \emph{per-device clipping} that clips the gradient of each model piece separately on its host device. Privately fine-tuning GPT-3 with per-device clipping achieves a task performance at $\epsilon=1$ better than what is attainable by non-privately fine-tuning the largest GPT-2 on a summarization task.

Deep 3D point cloud models are sensitive to adversarial attacks, which poses threats to safety-critical applications such as autonomous driving. Robust training and defend-by-denoise are typical strategies for defending adversarial perturbations, including adversarial training and statistical filtering, respectively. However, they either induce massive computational overhead or rely heavily upon specified noise priors, limiting generalized robustness against attacks of all kinds. This paper introduces a new defense mechanism based on denoising diffusion models that can adaptively remove diverse noises with a tailored intensity estimator. Specifically, we first estimate adversarial distortions by calculating the distance of the points to their neighborhood best-fit plane. Depending on the distortion degree, we choose specific diffusion time steps for the input point cloud and perform the forward diffusion to disrupt potential adversarial shifts. Then we conduct the reverse denoising process to restore the disrupted point cloud back to a clean distribution. This approach enables effective defense against adaptive attacks with varying noise budgets, achieving accentuated robustness of existing 3D deep recognition models.

尽管在各种应用中取得了突出的性能，但点云识别模型经常遭受自然腐败和对抗性扰动的困扰。在本文中，我们深入研究了点云识别模型的一般鲁棒性，并提出了点云对比对抗训练（PointCat）。 PointCat的主要直觉是鼓励目标识别模型缩小清洁点云和损坏点云之间的决策差距。具体而言，我们利用有监督的对比损失来促进识别模型提取的超晶体特征的对齐和均匀性，并设计一对带有动态原型指南的集中式损失，以避免这些特征与其属于其属于其归属类别群的偏离。为了提供更具挑战性的损坏点云，我们对噪声生成器以及从头开始的识别模型进行了对手训练，而不是将基于梯度的攻击用作内部循环，例如以前的对手训练方法。全面的实验表明，在包括各种损坏的情况下，所提出的PointCat优于基线方法，并显着提高不同点云识别模型的稳健性，包括各向同性点噪声，LIDAR模拟的噪声，随机点掉落和对抗性扰动。

本文提出了一个简单而有效的框架蒙版，该框架将新提出的掩盖自distillation纳入对比的语言图像预处理中。掩盖自distillation的核心思想是将表示从完整的图像提取到蒙版图像预测的表示形式。这种合并享有两个重要的好处。首先，掩盖的自我验证目标是本地贴片表示学习，这与视觉对比度的互补，专注于与文本相关的表示。二，掩盖的自我验证也与视觉语言对比符合训练目标的视野对比是一致的。视觉编码器用于功能对齐，因此能够学习本地语义从该语言中获得间接监督。我们提供了专门设计的实验，并进行了全面的分析，以验证这两个好处。从经验上讲，我们表明，当MaskClip应用于各种具有挑战性的下游任务时，可以在线性探测，填充和零拍摄中取得卓越的结果，并在语言编码器的指导下取得了卓越的结果。

基于图形的模型最近在人的重新识别任务中取得了巨大的成功，该任务首先计算了不同人之间的图形拓扑结构（亲和力），然后将信息传递给他们的信息以实现更强的功能。但是，我们在可见的红外人员重新识别任务（VI-REID）中发现了现有的基于图的方法，因为有两个问题：1）火车测试模式平衡差距，这是VI-REID任务的属性。两个模式数据的数量在训练阶段平衡，但推理极为不平衡，导致基于图的VI-REID方法的概括较低。 2）由图形模块的端到端学习方式引起的亚最佳拓扑结构。我们分析训练有素的输入特征会削弱图形拓扑的学习，从而使其在推理过程中不够概括。在本文中，我们提出了一种反事实干预特征转移（CIFT）方法来解决这些问题。具体而言，均匀和异质的特征转移（H2FT）旨在通过两种独立的设计的图形模块和不平衡的场景模拟来减少火车测试模态差距。此外，提出了反事实关系干预（CRI）来利用反事实干预和因果效应工具来突出拓扑结构在整个训练过程中的作用，这使图形拓扑结构更加可靠。对标准VI-REID基准测试的广泛实验表明，CIFT在各种设置下都优于最新方法。

我们提出了引导蒙面的自动编码器（bootmae），这是一种新的视觉BERT预训练方法。 Bootmae用两个核心设计改进了原始的蒙版自动编码器（MAE）：1）动量编码器，该动量编码器可作为额外的BERT预测目标提供在线功能； 2）试图降低编码器的压力以记住目标特定信息的靶向解码器。第一个设计的动机是通过观察到的，即使用预定的MAE提取特征，因为掩盖令牌的BERT预测目标可以实现更好的预训练性能。因此，我们与原始的MAE编码器并行添加了一个动量编码器，该编码器通过将其自己的表示作为BERT预测目标来引导预处理性能。在第二个设计中，我们将特定于目标的信息（例如，未掩盖贴片的像素值）直接传达到解码器中，以减少记住目标特定信息的编码器的压力。因此，编码器专注于语义建模，这是BERT预训练的目的，并且不需要浪费其在记住与预测目标相关的未掩盖令牌的信息时的能力。通过广泛的实验，我们的Bootmae在ImageNet-1k上获得了$ 84.2 \％$ $ $ $+0.8 \％$在同一预训练时期。 Bootmae还获得了$+1.0 $ MIOU在ADE20K上的语义细分和$+1.3 $ box ap，$+1.4 $+1.4 $ bask ap改进对象检测和可可数据集上的细分。代码在https://github.com/lightdxy/bootmae上发布。

现有的伪造检测方法通常将面部伪造视为二进制分类问题，并采用深层卷积神经网络来学习歧视性特征。理想的判别特征应仅与面部图像的真实/假标签有关。但是，我们观察到，香草分类网络学到的特征与不必要的属性（例如伪造方法和面部身份）相关。这种现象将限制伪造的检测性能，尤其是对于概括能力。在此激励的基础上，我们提出了一种新型方法，该方法利用对抗性学习来消除不同伪造方法和面部身份的负面影响，该方法有助于分类网络学习固有的常见歧视性特征，以进行伪造伪造。为了利用缺乏面部身份的地面真实标签的数据，我们根据来自现成的面部识别模型得出的相似性信息设计了一个特殊的身份歧视器。在对抗性学习的帮助下，我们的伪造检测模型学会了通过消除伪造方法和面部身份的影响来提取共同的歧视特征。广泛的实验证明了该方法在数据集内和交叉数据集评估设置下的有效性。

不同对象之间的闭塞是多对象跟踪（MOT）中的典型挑战，这通常导致由于丢失的检测到的对象导致较差的跟踪结果。多对象跟踪中的常见做法是重新识别出现后的错过对象。虽然重新识别可以提高跟踪性能，但是需要培训型号的身份的注释。此外，这种重新识别的做法仍然不能在探测器错过时跟踪那些高度遮挡的物体。在本文中，我们专注于在线多目标跟踪和设计两种新颖的模块，无监督的重新识别学习模块和遮挡估计模块，处理这些问题。具体地，所提出的无监督重新识别学习模块不需要任何（伪）身份信息，也不需要缩放性问题。所提出的遮挡估计模块尝试预测闭塞发生的位置，其用于估计探测器错过对象的位置。我们的研究表明，当应用于最先进的MOT方法时，所提出的无监督的重新识别学习与监督重新识别学习相当，并且通过所提出的遮挡估计模块进一步改善了跟踪性能。